chdkfandomcom-20200222-history
Adding support for a new camera
Acknowledgement: The original article on the subject was published in May 2007 by CHDK founder VitalyB. It remains a seminal text inscribed in the "virtual" CHDK Hall of Fame. *This page originally contained a copy of VitalyB's article that was later merged/replaced by the one from mweerden. *An older revision of this article with some helpful information is available from the page history, see Revision as of 22:51, 2009 January 22 Introduction This article is mainly intended for new porters to get started. Various other useful articles can be found here. Feel free to make changes when you encounter something incorrect, incomplete or unclear. Requirements Besides some knowledge of assembly and C, you will need to following things when making a CHDK port: * A dump of the Canon firmware (details) * Tools to disassemble the above firmware (e.g. IDA or others) * CHDK source + ARM compilation tools (see links here) * Some addresses for LEDs (discussed here) The workings of the CHDK To understand how the CHDK works we first must look at the Canon firmware itself. On startup, the firmware roughly executes the following: * If there is a locked memory card with a DISKBOOT.BIN file in the root: load, decode and run it. * Otherwise, start the normal booting process, which includes starting various tasks such as: ** Logging task ** Keyboard task ** Image-capturing task ** File-system task The CHDK adds another task and slightly adapts some of the existing task to make things work. It does this mimicking the normal boot process, substituting different behavior where needed. This includes adding hooks to the general task creation of the firmware, which allows it to start its own versions of certain tasks. All of this is possible due to the DISKBOOT.BIN execution. Put the CHDK code in DISKBOOT.BIN, and the firmware will execute it automatically. Note that this requires a FAT12 or FAT16 partition except for cameras running DryOS release 47 or later (starting in 2011) which can boot from FAT32 (details on making cards bootable). Also note that it is also possible to do the same by using a PS.FIR/PS.FI2 file and manually choosing to do a "Firmware Update". The details of what happens when DISKBOOT.BIN is executed are as follows. * The core CHDK code is copied to a suitable location in memory. * The camera is restarted with the CHDK code as new entry point (based on reset code from the firmware). * The CHDK adaptation of the boot process is run. It has the following differences from the firmware boot process: ** It adds task creation hooks. ** It starts the main CHDK task. ** Changes some details to ensure correct startup (i.e. prevent boot loops) * At the end of the above process, all (normal) tasks are created. Via the added hooks, the CHDK substitutes some slightly modified tasks. Overview To make a port for a new camera one has to search the firmware for code to base the adapted boot process and tasks on. Also, various camera-specific details and functions that the CDHK uses need to be implemented. The latter mostly entails taking already existing code of a port of a similar camera (i.e. your reference port) and finding the right constants for that code in the firmware. You typically find the origin of values/code in the reference port's firmware and try to use that to find something similar in your own. Finding a good reference port might not be trivial, so you might want to consult the CHDK experts on that one. Porting a camera consists of two main phases. The first phase is concerned about making the camera bootable under the control of CHDK code. When this phase is completed, the camera should be able to successfully boot a CHDK where all features are turned off. After that begins the second phase, where important constants and addresses must be determined from the firmware in order to make all features work. It is possible to activate one feature after the other, such as finding key codes to make the keyboard work or finding addresses of the framebuffers to enable graphical output. As we all know, it is good practice to document the code. For the CHDK port this mostly means that you should note where (and, in non-trivial cases, how) you have found certain values or pieces of code in the firmware. Also make sure it is clear which portions of assembly are straight from the firmware and which are changed, added or removed. Besides being useful when debugging, this documentation really helps people who are going to use your port as basis. We only cover the most important details here. Files or functionality that is not mentioned are typically straightforward to adapt (e.g. Makefile) or similar to other files/functionality discussed here. When you think you have done "everything", just go over each file to see if you have missed anything. Adding necessary files and folders for your camera The easiest approach is probably to take all the code of an already existing port as basis and adapt it file by file. To pick which port to use, try to find a CHDK supported camera released at about the same time as your camera. You will probably have to do some internet searching for that information. The DryOS revision is usually a good guide, and can often be found in P-ID (Table). Note that this is more important than having a similar model number or physical resemblance. Alternatively, ask for help on the CHDK forum - some ports are better than others. You'll need copies of the loader/ and platform/ directories, add(/copy) defines in include/camera.h and add the firmware dump named PRIMARY.BIN to platform//sub/ (N.B.: change the directory in sub to the right name for your firmware version). If you start by copying an existing port, it is essential that you update all the required values. If you copied something and aren't sure that it is correct, be sure to leave a comment to indicate this. Inappropriate copy/paste is a major source of bugs in new ports. As an alternative to wholesale copying of an existing tree, you can work through a reference port file by file and create the corresponding file in your new tree. As you go though each file, copy those parts that are clearly generic, and create any parts that are obviously specific to your camera, and mark any you are unsure of. You can help decided what is generic by comparing several existing cameras. In the initial pass of copying, you can leave out everything that isn't directly required to boot. Making your port compile To make your camera the default for make commands, create a file in the root of the source tree called localbuildconf.inc with lines like PLATFORM= PLATFORMSUB= This is not strictly required, but is assumed in the commands listed below. Refer to buildconf.inc for other build time settings. platform//sub//makefile.inc This is probably the first file you want to adjust. At first the most important values are: * PLATFORMOS: either vxworks or dryos (All cameras since around 2008 are DryOS, DryOS dumps have "gaonisoy" near the start) * ROMBASEADDR: should point to the start of the firmware. All known cameras use 0xFF000000, 0xFF810000 or 0xFFC0000, dependent on the size of the firmware. If you used the Canon Basic dumper to get your firmware dump, the start address will be recorded in cbdumper.log With this values entered, you can get many of the remaining values for a dryos camera using the finsig tool. To do this, run make rebuild-stubs The found values will appear in platform//sub// subs_entry.S if there are no errors, or stubs_entry.S.err if there were errors. In either case, it should contain many of the values mentioned below. * MEMBASEADDR: typically 0x1900 (for DryOS: the smallest address used in the loop above the occurrence of MEMISOSTART) * MEMISOSTART: points to the start of the memory pool used by the firmware (you can find this address at the end of the first piece of code in the firmware for DryOS) * NEED_ENCODED_DISKBOOT: Specifies the diskboot.bin encoding version. Leave undefined for no encoding, or see tools/dancingbits.c for possible values. All recent cameras need encoding. (Discussion). * KEYSYS not required, used to build PS.FI2 files for "firm update" loading. Now run make fir There might be errors due to undefined functions. For now, you can get rid of them by adding some dummy values to platform//sub//stubs_entry_2.S (e.g.NHSTUB(name,0x12345678) for function _name ; note the underscore which is not in the NHSTUB ). Note: In you should build make clean fir as the makefile dependencies are incomplete. platform//sub//stubs_entry_2.S The CHDK uses various functions that are readily available in the firmware. In order to do so the addresses to these function need to be known. The build system tries to guess (details) these addresses from the firmware dump (which are written to platform//sub//stubs_entry.S), but not in all cases does it find the right addresses. For this reason it is possible to add corrections to stubs_entry_2.S. These corrections should be of the form NHSTUB(name,address). Note that the macro NSTUB, as used in stubs_entry.S, should not be used; using NHSTUB ensures that the incorrect addresses found automatically are overridden. Any function with less than 100% match in stubs_entry.S should be manually checked. There might also be some functions that are completely missed by the automatic detection, or not included in the detection system at all. These will generate compilation errors that mention function names that start with an underscore. These should also be manually added to stubs_entry_2.S, but without the underscore (NHSTUB takes care of that). Boot loader The boot sequence The details of what happens when DISKBOOT.BIN is executed are as follows. #The core CHDK code is copied to a suitable location in memory. #The camera is restarted with the CHDK code as new entry point. #The CHDK adaptation of the boot process is run. It has the following differences from the firmware boot process: #* It adds task creation hooks. #* It starts the main CHDK task. #* Changes some details to ensure correct startup. # At the end of the above process, all (normal) tasks are created. Via the added hooks, the CHDK substitutes some slightly modified tasks. In this section, the boot loader will be created. This is probably the most complex step in porting, since it involves the implementation of various functions that will call one another. In order to not get lost, it is useful to implement the functions one after the other, using the following trick: A trick for keeping track During this process one can add debug statements that blink LEDs to indicate what the camera is doing. For example, you can use something like the following code (with 0x12345678 the address of the LED and N a reasonably large number such as 0x1000000): int i; *((volatile int *) 0x12345678) = 0x46; // Turn on LED for (i=0; i/entry.S The code in this file is the very first thing that gets called when DISKBOOT.BIN (or PS.FI2) is run. It typically just consists of the following code that calls the function in loader//main.c: mov sp, # mov r11, #0 b my_restart Some cameras seem to need some additional code before this piece for the automatic boot to work. See, for example, the SX10 port for the code snippet. As a test you can already add some debug code to turn on a LED here, but you can also just add the code to the beginning of my_restart() in main.c. If this doesn't work, you should make sure you have the right numbers in platform//sub//makefile.inc. loader//main.c Here we implement my_restart(). The standard code copies the reset code from the resetcode subdirectory to a safe place (at RESTARTSTART) and then calls that code. The reason it is copied is because when copying the CHDK core to its destination, which happens in the reset code, there might be situations where the destination and source (containing the executing reset code) overlap. Overwriting code that is running is not what we want. Note that the call to the reset code is a bit of a hack. Instead of directly calling the function copy_and_restart() we actually jump to the entry point of the reset-code binary. That is, we jump to the code in loader//resetcode/entry.S. The arguments we pass in the C code are "silently" passed on to the actual copy_and_restart() function. Loading the CHDK into memory loader//resetcode/entry.S Similar as the entry.S before, but this one calls the copy_and_restart() from loader//resetcode/main.c. Note the "silent" passing of arguments mentioned earlier (../main.c). loader//resetcode/main.c Here we implement the function copy_and_restart(). As arguments it takes the destination, source and size of the CHDK core. It first copies the core to the destination and then executes the slightly adapted reset code. You can typically locate this reset code in the firmware by using your reference port. Once you have found it, you must copy it (or make sure it is the same as the reference) and make sure you change the jump at the end to jump to the start of the core you just copied). Note that this last jump is to the start of the core which means it will execute the code in core/entry.S next. This code will call the startup() function in platform//main.c, which in turn will call the adapted boot code. Implementing the CHDK boot process When the camera is turned on without the CHDK, it executes the very first bytes of the firmware and thus booting the camera. In this section, you basically reimplement the first couple of functions that are called during the initzalization phase. This is necessary as you want to boot the original firmware, but want to hook a custom function later into the boot process. platform//sub//boot.c The adopted reimplementations of the first boot functions are located in this file. In essence you can just take the code from the firmware and copy the changes as made in your reference port. Once you have got the boot code ready, the camera should boot normally again. (Only difference is probably that a short press on the power-button gives you the review mode now.) For testing purposes you probably want to make sure that you have the standard startup() in platform//main.c; this function basically just calls the boot() function of boot.c. Besides the boot process itself, this file also contains the task-creation hooks, a function to start the main CHDK ("spy")task and one or two modified task (init_file_modules_task() and, if relevant for your camera, JogDial_task_my()). At first it is probably best to comment out the code of the task-creation hooks and CreateTask_spytask(). This way you can focus on getting the boot process working and uncomment the tasks as you finish them. The boot code is taken from the start of the firmware and the successive functions that are called from it. There a a number of changes that are made: * The task-creation hooks are added. * Power-button detection is improved; this makes sure that you only have to hold the power button for about a second to start in shooting mode. (Due to the more complex booting the firmware no longer correctly detects whether the power-button was pressed to start the camera.) * MEMISOSTART is replaced; the CHDK core is now load at that address so we must adjust the memory pool accordingly. * The CHDK task must be created. Getting the main CHDK task to run First thing to do now is to get the CHDK task to run. So uncomment the previously commented code and check that you get the CHDK boot screen now. It will take a few seconds because it is waiting for something that has to be done at a later stage. The relevant code of the CHDK task is in core/main.c. Also, you might need to fix some of the references in stubs_entry_2.S and stubs_min.S as well as the vid_*() functions in platform//sub//lib.c and debug_led() in platform//lib.c. At this point you can probably skip the tasks in this file. It is more useful to first get the keyboard to work properly for CHDK. (See platform//kbd.c.) Don't forget to uncomment the if statement with mykbd_task! Once you have spytask and kbd_task running, you should be able to use the PTP Extension to upload files, examine memory addresses, etc. Following the general keyboard task, you can add the JogDial_task (if your camera has a jog dial). Here you will have to add a piece of code (see, for example, the SX10 port). The purpose of this code is to be able to stop the firmware from reacting to the jog dial while one is in CHDK ALT mode. Finally, you must adapt init_file_modules_task(). Here an extra piece of code has to be added as well. This code is to support the autoboot feature on memory cards that are bigger than 4GB (by using two partitions). Also, after the call that takes care of this, we add a statement that signals the CHDK task it can start. This is done afterwards to make sure that this task uses the right partition. platform//kbd.c To get CHDK to detect button presses (and USB remote triggers), one needs to specify the bits that represent these. These bits are located in the physw_status array (defined in platform//sub//stubs_min.S). To figure out which bit is what, one can use the debug On-Screen Display (OSD) of CHDK. As the buttons probably don't work yet, you can force it to be always visible by uncommenting the commented call to gui_draw_debug_vals_osd() in core/gui.c and removing the if at the beginning of that function. Also make sure that in this function you display the physw_status array (one line for each of the three elements). With these changes you should be able to find each bit that corresponds to a specific button. Note that for the USB-power bit you can simple plug in a (connected) USB cable. The information obtained in this way should be put in keymap[] (except for the USB-power bit). Each entry is of the form { idx, KEY_NAME, mask } where idx is the index to physw_status and mask selects the bit (or bits) for the KEY_NAME button. That is, (physw_statusidx & mask) should be true if, and only if, button KEY_NAME is not pressed (not, because the bits are 0 if the button is pressed). The table should be ended with { 0, 0, 0 }. For the USB power you should define the macros USB_MASK and USB_REG (where the latter is the index for physw_status). Be sure to check the use of USB_REG as some code simple has the value hard coded instead of using the macro. Next you can define alt_mode_key_mask and the KEY_MASK? macros. The former, as static variable that can be set with a function kbd_set_alt_mode_key_mask(), is only required if you want the port to support a user-selectable "ALT" button (i.e. when you define CAM_ADJUSTABLE_ALT_BUTTON in include/camera.h and add the choices to gui_alt_mode_button_enum() in core/gui.h). Otherwise you can just make it a macro. For alt_mode_key_mask you should take the mask from keymap[] that corresponds to the button that should be used to enable the CHDK "ALT" mode (which should be KEY_PRINT to match the default CHDK configuration). The KEY_MASK? macros should be the or of all the masks of keys that correspond to the specific index (i.e. for KEY_MASK0 you should take the masks of keys that are represented in physw_status0). Besides these defines, you should also set SD_READONLY_FLAG. This is needed because autoboot only works if your memory card is locked and saving pictures only works if it is unlocked. Because the card lock is not a "true" lock, CHDK can fake that the card is unlocked by unsetting this flag. Note that there typically is no define to indicate the index of physw_status that is used, so make sure that the code you have uses the right index. Also note that you can get this bit with the debug OSD, but perhaps it is easier to find it in the firmware (see, for example, the comments in the IXUS 870 port). Choosing an alt button If the camera has a "direct print" button, that should be the default alt button. If it does not, the next choice should be the play button. If the port uses a non-standard alt button, it should be documented in the ports notes.txt and the cameras wiki page. Jog dial Finally, if your camera has a jog dial, kbd.c should implement the functions Get_JogDial() and get_jogdial_direction(). Don't forget to check that turning the dial in, for example, the CHDK menu feels intuitive. With this file complete, you should be able to actually use the CHDK. Of course only as far as the functionality that you have enabled at this point allows you. Additional functionality platform//sub//capt_seq.c In capt_seq.c we have two modified tasks: capt_seq_task() and exp_drv_task(). These are adapted to support saving RAW files, using a remote (USB) trigger and changing the exposure settings (including the addition of very long exposure times). exp_drv_task() is only required for the optional extra long exposure feature. platform//sub//movie_rec.c Here we have just one task. An adapted movie_record_task() to allow optical zooming during recording as well as changing the movie quality. platform//platform_camera.h Make sure all the defines in the been set correctly for your camera. This file defines both camera characteristics (like whether it has manual focus, adjustable aperture etc) and which CHDK features are implemented. platform//lib.c The function ubasic_set_led() is a bit of an odd one as it has no clearly defined behaviour (due to the fact that different cameras have different number and kinds of leds). The best guess seems to be to just try and approximate the behaviour as described in the documentation of the UBASIC set_led function. platform//main.c In main.c there are essentially two main things to adjust. First is fl_tbl[] (with CF_EFL and zoom_points). Depending on the number of different zoom steps your camera supports, there are two implementations. One has a table with all possible focal lengths (one for each zoom step; see, for example, the IXUS 980 port) while the other has just a small number of focal lengths and calculates the others through interpolation (see, for example, the SX10 port). To get the values for fl_tbl[] make a picture for each zoom step you put in the table and check the EXIF data for the reported focal length. The entries are in micrometers (i.e. mm*1000). To calculate the value for CF_EFL see the comments in the IXUS 980 port. Note that this value is typically multiplied by 1000 or 10000 to allow for accurate calculations using just integers (this factor is eliminated in get_effective_focal_length()). On some DryOS cameras the fl_tbl[] table can be found in the firmware and this data may be used instead of manually creating the table (as described above). The finsig_dryos program that builds the stubs_entry.S file will attempt to find the table in the firmware, based on the CAM_DNG_LENS_INFO setting in platform_camera.h. If the table is found in the firmware, details will be stored in the stubs_entry.S file which can be used in main.c. See the G12, SX30, IXUS310 or SX40HS ports for examples of this. To conclude this file, we have the get_vbatt_min() and get_vbatt_max() functions. Get the values by observing the voltages that CHDK reports over time (you'll have to change the display of percentage to voltage in the CHDK menu). You might want to take numbers that deviate a bit from the actually observed minimum and maximum, as the decrease over time is not linear and is likely to be higher at the limits. If your camera has a swiveling LCD screen, you must implement the screen_opened() and screen_rotated() functions here. Similarly, if the playrec_mode variable does not behave the same as expected by the generic rec_mode_active() function, you can implement your own version here. platform//shooting.c To fill aperture_sizes_table[] you can use the same method as for filling fl_tbl[] in main.c. In case your camera doesn't have an actual diaphragm, you just take pictures at the available zoom steps. Twice if your camera has an ND filter (once with filter and once without; either use the ND-filter option from the CHDK menu, if you have the right values in platform//sub//stubs_entry(_2).S, or find specific modes that make sure the filter is used or not). The entries are of the form { idx, av, string } with idx starting from 9, av the values of PROPCASE_AV and string a string representation of the aperture (the value of which you can get from the EXIF data). To fill iso_table[] you check PROPCASE_ISO_MODE for the different ISO values the camera allows you to select. Note that Auto gets index 0 and HI gets index -1. For shutter_speeds_table[] you essentially have to try various exposure times and see if there is a difference with others. If not, you know that the used exposure time wasn't really supported. To fill in modemap[], use the same procedure as key_map[] in kbd.c. Use the debug OSD with "Props" to show item PROPCASE_SHOOTING_MODE (see include/propcaseN.h for your camera) and change modes to see what value is shown. You might need to extend the enumeration in include/platform.h if your camera has new kinds of modes. For DryOS cameras, the signature finder will put the location of the canon firmware mode list in a comment in stubs_entry.S. On modern cameras, the canon mode list is an array of 16 bit numbers terminated by 0xFFFF. All the modes in this list should be available in the mode map, except those that correspond to C or C1/C2 customizable modes on high end cameras. The sig finder will warn if there is a mismatch between the modemap and the canon mode, and Lua/Scripts:Standard/Test/Setmode can be used to verify that mode setting works correctly. Finally, you'll have to find the right values for PARAM_FILE_COUNTER by using the debug OSD. You can easily detect them by taking a picture and noting which value changes. Note that PARAM_FILE_COUNTER is a big number as it actually contains multiple counters (see its usage in platform/generic/shooting.c; taking a picture will add 16 to the number as a whole. PARAM_EXPOSURE_COUNTER was used in earlier CHDK versions, but now it is not necessary. Exposure counter value is calculated by long get_exposure_counter() function (see core/shooting.c core/kbd.c Add your camera to the right definition of nTxtbl. This table should correspond with the zoom-step indices used for fl_tbl[] in platform//shooting.c. core/gui.c Check the #ifdefs at the top to make sure the keyboard shortcuts are appropriate for cameras key layout. Getting Your First Version Out If all went well, you should have a reasonably decent initial port by now. This is probably a good time to share your work with others and solicit feedback. A good way to do that is to post it on the CHDK forum. If there is already a "porting thread" for your camera then post an announcement about your new port with a link to the download service ( dropbox.com, box.com etc) where you are going to make it available. If no porting thread yet exists, go ahead and start one. You might also want to add a link to the wiki page for your camera. User feedback (and recognition of your work) are a big part of this next step. Keep your fingers crossed and try to stay positive about all feedback - positive, negative or (worse) indifferent. Submitting Your Port to the SVN Source Tree Once you are sure that your port is stable and reasonably complete and tested, you can submit it for inclusion in the CHDK autobuild process. Early ports can be submitted with Alpha status so that they are saved in the SVN repository but not released to the general public. More stable ports are typically released with Beta status. Beta status can be removed sometime later when things have stayed stable for while (how long is pretty much up to you). CHDK source code is maintained on the assembla web site. That's probably where you obtained your working copy of the code. To have your port added to that web site, you need to submit a patch file so that one of the CHDK developers with "commit" permission can add it to the repository for you. Typically, the preferred method to prepare a patch file is to use the diff function integrated into the svn software (the version control software used to maintain the CHDK source code). On Windows, this is typical Tortoise SVN . On Linux machines there are svn command line tools. There are also other packages for both systems. Patch files created in this format can then be feds directly into what is essentially the the Unix patch command to make all the changes automatically in the svn build tree. This method makes things a lot easier for the people who apply changes as they really don't have time to hand edit a long list of changes or wade through different patch file formats. To submit your patch file, post it on the CHDK forum here : link> |REQ| Adding new cameras, applying patches into trunk (with source code prepared) One final note : please be sure to include any port specifics such as non-standard buttons, known issues, missing/broken features in notes.txt in the platform directory of your port. Category:Development Category:Porting Links * Discussion about encoding/decoding PS.FI2 files [http://chdk.setepontos.com/index.php/topic,2995.0.html Forum: fi2 encoder/decoder] * How to extract the keys: For_Developers/fi2offsets Category:Development Category:Porting